Image related methods and systems

ABSTRACT

Digital watermark encoding—and associated registry transactions—are made transparent to consumers—performed as built-in features of common image processing operations, such as taking a picture, or printing a picture. In one arrangement, a user interacts with buttons and other controls of a graphical user interface on the touchscreen of a printer to author specific experiences that should be triggered by a hardcopy image—such as launching a related video, playing a recorded audio clip, displaying other images in a story narrative to which the hardcopy image relates, etc. The printer then attends to interactions with network infrastructure components needed to give the hardcopy print the user-desired functionality. In another arrangement, the content experience triggered by a printed image is authored automatically, e.g., by reference to data mined from the image&#39;s online context (e.g., Facebook photo album). A great variety of other features and arrangements are also detailed.

RELATED APPLICATION DATA

This application is a continuation-in-part of application Ser. No.13/117,026, filed May 31, 2011.

RELATED TECHNOLOGY

The present technology relates to that detailed in various of theassignee's previous patents and applications, including U.S. Pat. Nos.6,122,403, 6,590,996, 6,947,571, 6,952,485, 6,970,886, 7,010,144,7,450,734 and 7,480,393; patent publications 20060062426, 20070156726,20070070429, 20100150434 and 20100048242; and patent application Ser.No. 13/007,372, filed Jan. 14, 2011.

The present technology expands, in some respects, on technology detailedin the above-noted documents. The reader is presumed to be familiar withsuch previous work, which can be used in implementations of the presenttechnology (and into which the present technology can be incorporated).

TECHNICAL FIELD

The present technology concerns improved methods and systems forinteracting with imagery, such as by providing printed photographs withenhanced network functionality.

BACKGROUND AND SUMMARY

In earlier patents, the assignee detailed how imagery can be encodedwith hidden digital watermark data. Such data can be used, e.g., to taga photograph with the name of the photographer and other identificationdata. This identification data is used, in some embodiments, to link toassociated web destinations. For example, a photographer may watermarkphotographs with an identifier which, through an intermediate registry,links to the photographer's web site.

Digital watermark technology is also used to encode printed news andentertainment media, such as magazines and newspapers, withsteganographic identifiers that point to associated network resources.Thus, for example, a smartphone may capture imagery from a news storyphoto, decode the watermark identifier, pass it to a registry, andreceive a link to a web site devoted to late-breaking developmentsrelating to that news story.

While such technology has been adopted by many commercial enterprises,adoption of digital watermarking technology by consumers, per se, hasbeen limited. Previously, there has been no system for watermarkencoding imagery, and for interacting with an associated registrydatabase, that has been simple enough for widespread home use.

In accordance with one aspect of the present technology, digitalwatermark encoding—and associated registry transactions—are madetransparent to consumers—performed as built-in features of common imageprocessing operations, such as taking a picture, or printing a picture.

In an illustrative embodiment, a user particularly defines networkexperiences that a hardcopy image is to invoke. For example, the usermay interact with buttons and other controls of a graphical userinterface on the touchscreen of a printer to author specific experiencesthat should be triggered by a hardcopy image—such as launching a relatedvideo, playing a recorded audio clip, displaying other images in a storynarrative to which the hardcopy image relates, etc. These choices can bemade at the time of printing, or the printer can be used as an interfaceto establish or adjust such network experiences after printing. Theprinter then attends to interactions with network infrastructurecomponents needed to give the hardcopy print the user-desiredfunctionality.

In other embodiments, the content experience triggered by a printedimage is authored automatically, e.g., by reference to data mined fromthe image's online context (e.g., Facebook photo album). In still otherembodiments, hybrid approaches can be employed—with some responsesdefined automatically, and others defined by user action.

The foregoing and additional features and advantages of the presenttechnology will be more readily apparent from the following detaileddescription, which proceeds with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 show components that may be used in certain embodiments of thepresent technology.

FIG. 2 shows a user interface through which a user can define a contentexperience that will be presented to other users who capture imageryfrom a printed object.

FIG. 3 shows a block diagram of an illustrative smartphone.

FIG. 4 shows a block diagraph of an illustrative multifunction printingdevice.

FIG. 5 details an illustrative usage scenario, involving a series ofinteractions rooted in an initial photo image.

FIG. 6 show shows a sample social networking site web page about theinitial photo image of FIG. 5.

FIG. 7 shows a further usage scenario, continuing where FIG. 5 left off.

FIG. 8 shows a data structure/table maintained by an illustrativeregistry, detailing information related to the initial photo image ofFIG. 5.

DETAILED DESCRIPTION

With the proliferation of camera-equipped smartphones, the number ofphotographs taken by consumers has surged. Users can readily post theirphone-captured imagery to Facebook and other online sites, and richnarratives and content experiences can develop around suchpostings—e.g., with comments and links contributed by the user's socialnetwork friends. When such photographs are printed, however, this wealthof additional information is lost; just the bare pixels remain.

In accordance with one aspect of the present technology, a picture doesnot become divorced from all of its extra information by printing.Instead, the photograph is printed with information that automaticallyenables linking back to its abundance of online annotations andassociated content.

FIG. 1 shows an illustrative environment 10 including a mobile phone 12,a printer 14, and a cloud 16 of online resources. For example, the cloudencompasses Facebook and other social networking sites 18. Similarly, itincludes online image repositories 20, such as Flickr, Picasa, OFoto,SnapFish, Google Images, etc.

Consider a Facebook photo page. While each such page includes an image,what makes it really interesting is the metadata around it: who postedit (Ethan); when it was taken (Nov. 20, 2010); and where it was taken(lat/long, or city name). Still more interesting may be the tags andcomments authored by the image owner, social network friends and otherviewers. Such text annotations may detail the circumstances when thephoto was taken (e.g., party in dorm after the college won the bigfootball game against the cross-state rival team); who was there (Ben,David, Maria, Christie, Ryan and Mitch—each labeled); what happenedbefore and after the picture was taken (a cross-state friend calledMaria, and the roomful of people sang the team's fight song to her onspeaker phone); online discussions triggered by comments relating to thephoto—but then diverging into ancillary topics, etc., etc. (Suchinformation may be termed crowd-sourced commentary.)

(Also relevant may be the temporal context of the image relative toother stored images in a Facebook collection (album), e.g., with a groupof pictures all relating to a common event. Relatedly, visual similarityof one image to a second image may allow metadata stored in connectionwith the second image to be used with the first. These two attributesare examples of “semantic proximity”).

A visitor to this Facebook page may print the picture for posting on adorm bulletin board. When the picture is printed, it includes a hiddendigitally-watermarked identifier that serves as a link back to theoriginal Facebook page. When another student later views the printedpicture on the bulletin board using a mobile phone, the phone loads theFacebook page so the student can review all of the crowd-sourcedcommentary.

By such arrangement, the richness of the photograph's network metadatais preserved—even after printing. Moreover, the online informationpresented via the mobile phone is real-time information, including thelatest crowd-sourced annotations. Thus, the printed photo may yield adifferent experience today than yesterday, due to the addition of morecommentary.

Particular implementations can take various forms. The image can bewatermarked when it is captured, when it is uploaded from the cameradevice to another repository (e.g., to a personal computer); when it isposted at Facebook; when it is first viewed at Facebook; when it is sentto print, etc. Similarly, the watermarking operation can be performed byany device in the data flow (or distributed between plural suchdevices), e.g., the digital camera or mobile phone 12; the personalcomputer; a router in a home network; a server at Facebook; anotherserver in the cloud (e.g., a Google print server 22); the printer 14itself, etc.

The identifier that is watermark-encoded into the printed picture is 48binary bits in a particular embodiment. Such a payload allows for theunique identification of many trillions of images. Larger or smalleridentifiers can of course be used.

Alternatively, the identifier can be of another form, such as a DigitalObject Identifier (DOI) or other Corporation for National ResearchInitiatives (CNRI) “handle,” or an IPv6 identifier (or part thereof).

In an exemplary embodiment, the identifier is issued by an “issuingauthority,” which may be a party such as Verisign, a domain registrar,ICANN, etc. The issuing authority can interact with one or more registrydatabases 24 that associate the identifiers with other information, suchas links to relevant sources of metadata, etc.

When an image is first watermarked, a request is typically made to anissuing authority for issuance of a new identifier. The authority issuesan identifier, and logs related information in the registry database.This related information can include linking data, if available.Alternatively or additionally, linking and other metadata may be addedor revised later—so-called “late binding.” (In some embodiments, theregistry may consult rules to determine who may request identifiers, oradd information to the database.)

When the watermarked image is thereafter imaged by a mobile phonecamera, or is otherwise accessed, the registry database is againcontacted—this time to retrieve one or more items of metadata. (As withimage watermark encoding, the watermark decoding operation can beperformed by any device(s) in the system.) A URL link obtained in suchfashion can be activated automatically, or at a user's election, toobtain crowd-sourced commentary and other metadata related to the image.

First Particular Embodiment

Increasingly, printers are web-connected and offer browsingcapabilities. The Hewlett-Packard Photosmart eStation Printer Seriesmodel C510, for example, includes an Android-based tablet computer forits control panel, providing full tablet computer functionality.

The C510 printer includes software (an HP tablet “app”) designed toprint photos from Facebook—excerpting from the web page just the pixeldata for printing, and automatically sizing the image for the availableprint substrate, etc. (Similar functionality is available from“PrintApps” that run on WebOS on a variety of Hewlett-Packard printers.)In accordance with this aspect of the present technology, when a userinstructs the printer to print a Facebook image, the Facebook apprunning on the printer 14 contacts the registry 24 and requests an imageidentifier. With the request the printer provides the registry a link tothe Facebook page from which the image is being excerpted.

The registry responds by returning a newly-assigned identifier to therequesting printer, and logging the received link in a new databaserecord. (Storing associated metadata with the identifier at the time ofits issuance may be regarded as “early binding.”) Meanwhile, the printerretrieves the user-identified image data from the Facebook page. Theprinter then resizes the received image data for the requested printing,digitally encodes the identifier into the resized image, and prints therequested photograph in hardcopy form.

The resulting printed photograph is logically bound to the associatedFacebook webpage, through the record in the registry database.

Note that there was no express user involvement in the metadata linking.The user simply printed a photograph, and the watermark-relatedoperations ensued automatically. (Such functionality is normally enabledby default, but may be disabled by a parameter in the User Preferencessetup of the printer, if desired.)

Second Particular Embodiment

One of the ways that the second embodiment differs from the first isthat the second embodiment allows the user to particularly tailor theexperience that the printed image is to invoke (whereas, in the firstembodiment, the experience is authored automatically, e.g., by a link tothe original Facebook page).

In a particular implementation of this second embodiment, the printercontrol panel (e.g., the touchscreen of the Hewlett-Packard C510printer) presents an authoring user interface 30 (FIG. 2) that allowsthe user to specifically define a multimedia presentation includingtext, audio, imagery, and video.

The top portion of the user interface (UI) 30 allows the user toindicate—by tapping check boxes—whether the hardcopy photograph shouldtrigger presentation of audio, image/video/web page (selected bytapping), and/or text. If audio or image/video is selected, a dialog boxopens thru which the user can navigate to the desired content, e.g., onthe internet, or locally. If text is selected, a touch-screen keyboardappears through which the user can enter the desired text message. (Ifthe user navigates to local content, the mobile phone automaticallycopies such content from the local address to an online repository, suchas the user's MobileMe, Picasa or SnapFish account, to allow access bythe others.)

The illustrated UI allows the user to script a presentation that evolveswith time, by stating how long (in seconds) the just-indicated contentshould be presented, before transitioning to other content. If thisoption is selected, a middle portion of the UI becomes active (it may begreyed-out unless/until the user enters a time interval in box 32).

The middle portion of the UI is similar to the top portion—here allowingsome of the audio, image/video/web page, and/or text from theearlier-detailed presentation to continue, while allowing other of suchcontent to be changed.

If the user taps a check box 34, the UI scrolls up to allow entry ofstill additional presentation details, e.g., by replicating thejust-discussed middle portion for entry of data about a third temporalphase of the presentation. In like fashion, the user can author fourthand further phases.

At the bottom of the screen is a check box 36 asking whether the userwants to store the just-entered data as a template—which can then berecalled as a starting point when scripting the content experience forother hardcopies of the user's photographs. If the user taps this box, adialog box opens asking for the user to name the template. Thereafter,when the UI 30 is presented, the user is first asked whether thepresentation it should be pre-configured in accordance with one of theuser's stored templates.

The UI 30 may be invoked at the time of initial printing. Alternatively,it may be invoked post-printing, to then define (or edit) the associatednetwork experience invoked by the hardcopy print.

(It will be recognized that the FIG. 2 arrangement is exemplary only.Implementations are likely to differ in form and vary in function. Whiledisclosed in the context of a printer, a similar authoring capabilitymay be provided by software on the user's mobile phone.)

Although not shown in FIG. 2, the user may further tailor the onlineexperience so that it varies depending on the particular person viewingthe photograph. If the user's roommate views the picture with his mobilephone, one set of content may be presented. If the user's mother viewsthe picture with her mobile phone, a different set of content may bepresented. (If the user's mother's native language is Spanish—asindicated by profile information provided by her phone, or by anassociated online profile—then any text included in the presentation maybe machine-translated into Spanish.)

In common practice, the experience invoked by a hardcopy print isinitially be defined automatically by the system, without any creativeuser involvement (as in the first embodiment). With that experience as astarting point, the user then employs the UI 30 of FIG. 2 to edit thisinitial experience—yielding a user-customized experience.

Third Particular Embodiment

In another embodiment a user sends an image to print from a mobile phone12. The image may have been just-captured (and stored only on thephone), or the user may have navigated to it online, using a phone webbrowser or other phone app software.

One way to print from a mobile phone is through use of Google's CloudPrint service 22. This service is provided by Google to allow any web,desktop or mobile application to print to any printer, using Google asan intermediary.

In particular, Google provides a web-based, common print dialog (web UI)or API, which the application uses to send a print job to a designatedprinter. Printers are setup in association with Google accounts, and maybe shared with other users (akin to Google Documents). The capabilitiesof each printer are stored by Google, and are reflected in the UI, e.g.,enabling users to set print preferences. Software executing on Googleservers pre-processes imagery as would a conventional printer driver,rendering the received data into image form, and providing signalsappropriate to the receiving printer. (The Google service mayadditionally perform operations such as color correction, down-sampling,and adding title overlays such as date/time/GPS location.) Even old,legacy printers can be accessed through this service, by using anattached computer as a proxy server. (In such case, the phonecommunicates with Google, which communicates with the computer, whichcommunicates with the printer. With more modern computers, Googlecommunicates directly with the printer—using the printer's own IPaddress.)

Consider a user who is on vacation in Italy, and who has used asmartphone to capture an image of a traveling companion posed as ifsupporting the Leaning Tower of Pisa. The smartphone automaticallyassociates a variety of metadata with the image using EXIF header data,e.g., date, time, GPS location, exposure and flash parameters. Otherparameters may also be stored automatically, such as sensor data fromthe instant the image was captured (accelerometer, gyroscope,magnetometer, etc.) The user may contribute still other data—such as atext caption to recall other events of the day. The name of the personin the photo may be added by the user, or may be annotated automaticallyby facial recognition software in the phone, or invoked by the phonefrom a cloud-based service. Still other context information may beharvested from other data sources. For example, the user's personalcalendar may have a stored itinerary for the vacation, including atextual description of the day's activities, restaurants to be visited,and the hotel where the travelers are lodged. Likewise, GPS coordinatescan allow identification of other images taken from the same locations(e.g., on Flickr), and metadata from those images may be extracted foruse with the user's images. (Such techniques are further detailed inpublished application 20100048242.)

The user may wish to print a copy of the image at a public printer atthe hotel in the evening, and mail it as a postcard to friends who areworking back at the office. The printer is accessible to guests throughthe Google Cloud print service. The user presses “Print.”

Software in the phone responds by transmitting the image, and some orall of the associated metadata, from the phone.

The metadata is routed for archival storage at a cloud service, such asSnapFish, Picasa, or an Apple MobileMe account—with or without the imagedata. The phone may send the metadata directly to such destination, orthe Google Cloud Print service can attend to this task.

The phone sends the image data to the Google Cloud Print service,together with an indication of the destination printer. In thisembodiment, the Cloud Print service solicits a watermark identifier fromthe issuing authority 24, and also provides a link to the archivalrepository where the metadata is stored. The issuing authority issues anidentifier, and stores the identifier in the database—together with theprovided link to the associated metadata.

Upon receipt of the watermark identifier, the Google Cloud Print serviceperforms a watermark encoding operation as part of its print jobpreparation, so that the print data sent to the hotel's printer includesthe embedded identifier.

Again, the result is the printing of a hardcopy photograph that islinked to all of the associated metadata. Again, this functionality isprovided without any user intervention or expertise; all was automatic.And this time such functionality is achieved without any intelligence inthe printer. (Indeed, old legacy printers can be used in this fashion.)The watermark encoding is performed in the cloud—as the data is en routefrom the phone to the printer.

Fourth Particular Embodiment

The foregoing two embodiments provided imagery from the “cloud” to theprinter. But this need not be the case. In a further embodiment, themobile phone 12 provides imagery directly to the printer 14—such as byWiFi, Bluetooth, etc.

As in the above-described second embodiment, metadata for the subjectimage is desirably copied to an archival site, such as an online photorepository—in this case by the phone 12. The phone then sends a requestfor an identifier to the issuing authority (together with address datafor the archived metadata), and then watermarks the imagery with thereceived identifier. The watermarked imagery is then transmitted, e.g.,wirelessly, to the printer for printing as a hardcopy print.

Alternatively, the watermark encoding can be performed by the printer.In this case, a pointer to the archived metadata is provided to theprinter along with the image data, and the printer attends to obtainingan identifier from the registry 24, and to providing a link to thearchived metadata for storage in the registry database.

Another variant embodiment splits the identifier-procurement andidentifier-embedding operations between the phone and the printer. Thatis, the phone attends to interaction with the registry (requesting anidentifier, and submitting metadata linking information for storage inthe registry database). The phone transmits the watermark identifier(received back from the registry) to the printer. The printer thenembeds the provided watermark identifier in the provided imagery, andproduces a hardcopy print.

Authentication

Policy and rules may be used in determining when, and who, can performbinding of a watermark identifier to network assets (either at time ofwatermark embedding, late-binding or changing of binding.) The act ofdoing so can be considered an extension of the authoring act, and assuch the author of the image may want to preserve the right to do thebinding themselves or explicitly delegate the right to others. Oneaspect of enabling policy and rules is the creation of roles within thesystem wherein each component or actor in the system has a verifiableidentity.

Many devices have unique identities (MAC address, UDID on mobiledevices, etc.) that can be utilized for this purpose. User identity canbe created and managed directly through existing services, such asMicrosoft Passport, or by leveraging existing identity mechanisms suchas a user's Facebook account, Gmail login, etc.

In one such scenario, an assigned identifier (watermark ID) within theregistry database can also contain one or more fields that link theidentifier to a user identity provided by external services. Once such alinkage is made, the registry database can rely on these services tomanage the authentication process, thus allowing the registry databaseto provide binding services per stored policies or rules that define theroles of the various actors and system components.

Such identity can inform how all embodiments defined herein areimplemented to reflect the wishes of the users and provide more securityto the system.

For the first embodiment, the authentication of the user is implicit, asthe user had to have access to their Facebook page in order to retrievethe image and print it locally. In such a case, the registry databasemay be setup to singularly trust requests originated from authenticatedusers of Facebook. While simplifying the embodiment, the reader willappreciate the limitations this would place on adoption of such asystem, by requiring all users to have Facebook accounts.

Setting aside this limitation for a moment, imagine a future embodimentof the HP PrintApp software in which the user is able to access theirFacebook page generally as opposed to only a specific photograph withintheir album. In such circumstance, the user might just as likely selecta picture that originated from a friend's page to print.

Currently there is no prohibition against printing any imageryencountered within Facebook, or for that matter within many other socialnetworking or photo sharing sites. The act of binding a photograph to adigital destination, however, is akin to creating a digital identity forthe photograph—something the author may wish to control.

At the point that the user prints a photograph of which they are not theauthor (proprietor), the request for a Watermark ID and binding of saidID to a service may be subjected to rules defined by the author.Enforcement of such rules may occur at different locations within thesystem.

The Facebook print app may, upon the user's request to print a friend'sphotograph, notice that the user is not the author of the photograph andcompletely forego the request for an identifier, watermarking andbinding operation. A more evolved embodiment involves the print appquerying Facebook for what rules to apply in such a scenario, with therules likely informed by the author's desires for the photograph. Inanother embodiment, the author may grant permission to the user to doso, or might require that the watermarking and binding to be done to aspecified destination (URL) as a condition of embedding.

As an example of the last scenario, the author may want to ensure thatwherever their pictures travel, that they are always connected back tothe appropriate Facebook page that contains compelling metadata aboutthe picture. A grandparent may see a picture on their Facebook page oftheir Grandson at a first baseball game of the season, authored (picturetaken and uploaded) by their daughter, and decide to print the picture.The daughter, knowing the grandparents' keen interest their grandson'sbaseball stats for the year, may enforce a rule that—when printed—thepicture should always link to the grandson's stats page that containsadditional details of the season, such as the game schedule, commentsfrom teammates, or additional media (pictures and video).

Returning to the enforcement of such rules, it should be made clear thatenforcement may occur at the print app as described above, withinFacebook itself, at the registry database, or even by a third partyrules and policy service.

In the third embodiment, the Google print service may act as the sourceof user identity, and enforce rules related to the wishes of the authorof the photograph. A user with a smartphone that has navigated to aPicasa photo album, for which they are not the owner, may identify apicture that they would like to print. At such time, the Google printservice may inquire as to the policy set by the author of the photograph(the author being identified through the Google user ID associated withthe creation of the photo album). Similar to the Facebook example above,the Google Print Service may then enforce the rules directly.

In both examples, pre-existing user identification mechanisms providedby Facebook and Google respectively are utilized, indicating that usersmay have multiple identities provided by different services, all ofwhich need to be understood by the registry database or systemresponsible for enforcing policy.

An approach to managing a user's multiple potential identities is tohave a central authentication service associated with the registrydatabase. At initial setup, the user links the user's identities fromrelated services that may issues a request for an identifier, or thatmay wish to do binding on the user's behalf. Doing so would allow theregistry database to be independent of any specific user identityprovider as originally described above.

Once set in place and enabled with well understood API's, all thescenarios and embodiments defined herein can leverage policies and rulesto automate and simplify the requisition and binding of IDs.

Device Domain Construct of Identity

Another embodiment for ensuring that the registry database is acting inaccordance with the author's wishes is to use a domain construct whereonly trusted devices within the domain of devices owned by the authorare allowed to make requests.

A user may associate devices with their identity as managed by thecentral authentication service defined above. The devices may consist ofphones, printers, laptops, game systems, internet connected TVs, etc.

Once associated, the device ID can be used to inform when and how rulesshould be applied. As an example, a simple rule an author might setup isthat only requests from their trusted devices are able to requestidentifiers and perform binding.

If the author navigates to the author's public Picasa album from theirphone, uniquely identified by its UDID, then the author has the abilityto perform watermarking and binding. If any other phone (with adifferent UDID that is not within the author's domain of registereddevices) attempts to do so, watermarking and binding will not beallowed.

General Comments

As just detailed, the system may be configured so that different usersenjoy different privileges. For example, if an image was captured by themobile phone of User A, then the system may allow only User A to authora corresponding content experience for that photograph. User A may usethis privilege to define an initial content experience, but mayexpressly authorize the system to accept edits from social network“friends” of User A. User A may further specify that certain elements ofthe original online experience should not be changed. User A may alsorequire the system to seek approval from User A before any edits made bysocial network friends are given effect.

Alternatively, User A may not impose any limits on who can author oredit content experiences for images captured by User A. Naturally, agreat number of other control and privileges attributes may beestablished.

Typically, such conventions are recorded in a Preferences file (e.g.,stored in the registry database) that User A can edit. Such Preferencefile is initially configured with default parameters.

Similarly, the system may be configured so that different contentexperiences are rendered to different users. User A may define two ormore classes of audience members—and specify, e.g., that certain onlinecontent should be omitted when responding to certain classes, or thatdifferent content be substituted. User A may further specify that theonline content experience may depend on parameters determined—at thetime of rendering—about the viewer (e.g., native language, as indicatedby online profile accessed using an ID provided from the user's device).

Some classes may be defined by lists of particular individuals. Othersmay be defined by one or more attributes. Exemplary classes include: (1)self; (2) Kate Middleton; (3) social network friends having an agewithin five years of User A's age; (4) other social network friends; (5)everyone else.

System control/privilege attributes and responsive behaviors are thuscommonly based on confirmation of identity, e.g., who a user is, whatdevices the user customarily uses, what images were captured with thatuser's devices, etc. A variety of cryptographic technologies are knownto establish reliable online identities, to associate particular userswith particular devices, and to associate particular content withparticular users/devices. The artisan will be familiar with suchtechnologies (which include, by way of example, hashing, public/privatekey encryption, digital signatures, etc.).

Some such technologies allow users to be authorized as members of acertain class, while still keeping their exact identities anonymous.Anonymous viewers may be regarded as a distinct class by User A, who maytailor content to be rendered to such users accordingly.

Similar user identification/authentication may be required in order toaccess certain of the online content presented when a hardcopy print isimaged. For example, if a hardcopy image links to crowd-sourcedcommentary about the image from a user's Facebook page, then viewersneed access to that Facebook page. Facebook has protocols allowinglimited access to Facebook-posted content without signing-in with aFacebook account, and such protocols can be employed to enable publicaccess. Alternatively, the software may be arranged to that the viewer'smobile phone automatically presents Facebook credentials to retrievesuch content, so that Facebook authorizes the needed access.

In all of the embodiments detailed herein, late binding of the metadatacan be employed. That is, the registry can issue a watermark identifierwithout storage of associated metadata in the database, and suchmetadata can be associated with the identifier at a later time.(Metadata can always be updated later, as old data needs revision, or asother information becomes available.)

In some embodiments, the user is queried—at the time of printing—whetherthey would like the hardcopy photograph to link to an onlinedestination. Preference settings stored in the printer, phone, or GoogleCloud Services account, may establish—as default behavior—that such aquery should be presented whenever a photograph including a watermark isto be printed, if the watermark is found to have no associated linkinginformation stored in the registry database 24. (The printer can contactthe registry database to check for the absence of associated linkinginformation, or the mobile phone, a Google server, or another device canperform such checking.) Such a query is typically presented to the uservia the user interface of the phone, or via the user interface of theprinter.

Many printers have memory card readers, for receiving images stored bycamera devices to such cards. The embodiments detailed herein cannaturally use such physical media movement to transfer data from cameradevices to printers or other devices. In such embodiment, the memorycard may contain rules and data that instruct the printer on how towatermark, print, and upload the photos and associated metadata to thecloud. Such information on the memory card may be prepared either by thecamera (e.g., firmware), or by a computer.

While embodiments detailed above reference the printer undertakingcertain actions, it will be understood that it is typically software onthe printer that is so-acting. While the HP PrintApp is one example ofsuch software, there is no reason that the detailed functionality cannotbe performed by other software, such as firmware, drivers, etc. Indeed,such functionality can be implemented in hardware, if desired in aparticular embodiment.

Most of this specification contemplates that metadata associated with aprinted photograph is displayed on a screen of a mobile phone or otherdevice that images the photograph. In some embodiments, however,printing of the photo could include printing of some or all of theassociated metadata. Such printing could either be on the photo, or tothe side/beneath/around. (For example, an 8.5″×11″ page may be printedwith a 4″×6″ in the center, and comments presented in the surroundingborder.) The particular layout can be defined by the user, e.g., with anauthoring UI akin to UI 30 of FIG. 2. By such arrangement, a user couldprint a photograph and include all the Facebook comments that friendsauthored—for enjoyment even without a mobile phone.

Online authoring tools are known from other services, such as Hallmarkand Evite. These web sites provide dashboard UIs that allow a user tocraft a particular web displays that are presented to visitors of theuser's online cards/event invitations. Similar constructs and principlescan be employed in authoring tools for creating online experiencesassociated with hardcopy printed documents, e.g., employed in the UI 30of FIG. 2.

In some instances, the established Hallmark and Evite authoring toolsare employed by a user to author a particular online experience. Thebehavior of a watermarked hardcopy document can then be set, by a linkstored in a routing table at the registry, to direct viewers of thedocument to the user-authored Hallmark or Evite online experience.

In like fashion, other online destinations and businesses may beinvolved in the “payoff” delivered to viewers of watermarked imagery.After sensing a hardcopy picture, for example, the registry may havedata causing the viewer's mobile phone to present an audio/image/textexperience, after which the phone concludes by linking to an onlineflorist site, where the viewer may make a purchase, or redeem anelectronic coupon that is delivered as part of the online experience.

As still another example, a printed invitation to a neighborhood bookclub gathering may cause viewers' phones to link to the sender'sDelicious social bookmarking website. There, viewers can tap on links todifferent book reviews, author biographies, etc., chosen by the sender,relating to that month's book. Thus, watermarked documents can serve asa portal to a variety of web destinations, which the document proprietorcan customize to deliver a desired experience.

Digital watermarking technology, and its use to associate networkresources with imagery, is detailed, e.g., in the patents referenced inthe Related Technology section of this disclosure. In some embodiments,a digital watermark encoder (e.g., in a printer or elsewhere) caninclude an image classifier to identify different classes of images,e.g., text (dark on light background); text (light on dark background),tinted content, natural color imagery, limited color palette imagery,black and white imagery, etc. The watermark encoder can apply differentwatermarking parameters (e.g., embedding strength), or methods (e.g.,chrominance versus luminance watermarking), in correspondence to theparticular class of imagery identified by the image classifier.

Additional functionality can be provided, in some embodiments, byproviding a thumbnail of the image, and/or an image fingerprint, to theregistry (or an associated data structure) for storage. Such data allowsthe registry to discern whether an identifier has already been assignedfor an earlier instance of that particular image—even if one imagediffers from the other in small respects, such as by resizing, or withcolor correction. If an earlier instance is noted, the registry mayreturn the identifier earlier-assigned back to the requesting device.Alternatively, a new identifier can be issued, and the registry can makean entry in its database record for one or both instances of the image,to note that another instance of that image was issued ID such-and-such.

If a device (e.g., a printer) is involved in printing multiple copies ofthe same image, at one time, it may request an identifier for the firstcopy, and then use the same identifier for all identical copies in theprint job—without further involvement of the issuing authority. (Thedevice may inform the issuing authority of its action, i.e., thatmultiple copies of a given image with the same watermark payload havebeen printed.)

While a table structure (e.g., a routing table) can be employed toassociate one or more URLs (and/or other data) with each assignedidentifier, other implementations can employ semantic assertions, suchas RDF triples. For example:

98D28D478C34 HasOwner JohnSmith458 98D28D478C34 HasTextMetadata Milo

98D28D478C34_Milo HasURIhttps://www.facebook.com/text.php?fbid=1015025983048134098D28D478C34 HasImageSetCompanionhttps://www.facebook.com/photo.php?fbid=1015025983048134

98D28D478C34 HasImageFingerprint 69F977CDE1FCB6A9EAD8C2 98D28D478C34HasIssuanceDate May 24, 2011 98D28D478C34 HasSubmitterPublicKey6D1CC27C49

Metadata associated with an image can include other images (or videos)that are related. For example, if a user uploads a series of images toan online repository, and all the images were captured on the same date,and within a defined interval of time (e.g., two hours), they may beassumed to relate to each other. Thus, metadata for one image mayinclude the other images in the set (and their respective metadata).

Similarly, if images were captured at the same geographic location,relationship again may be inferred. (A vacationer may return to OldFaithful repeatedly, years apart. Yet by the common geolocation, suchimages can be related.)

Images may also be found to relate to each other due to common subjectmatter, as indicated by visual similarity. For example, facialrecognition may find that two images in a user's Facebook photocollection depict the same person—despite being stored in differentalbums, and have been taken at different places and times. Other objectrecognition techniques (e.g., image fingerprinting) can be employed tofind common subject matter between images. Again, once related imagesare found, metadata can be shared between them. (For more on this topicsee, e.g., the assignee's patent publication 20100048242.)

As a particular example, consider two families who take a vacationtogether. The vacationers may have, between them, multiple mobile phonecameras, digital cameras, and video cameras. Several members of thegroup may each take pictures of the same object, e.g., a sailboat onwhich they take an afternoon trip. Each person may post their ownpictures/videos to their own respective Facebook pages (or other onlinerepositories). Yet by the common geolocation, and common date/time,these pictures can all be associated together. Further, by use of imagefingerprinting, or other object recognition technologies, the photos ofthe sailboat may all be identified. A merged collection of sailboatpictures may thereby be virtually assembled.

The collection may be reviewed—by a member of the group, orautomatically—to select a representative assortment of good images forprinting in a booklet form, as a trip memento. (An automated techniquecan consider technical merits of exposure and focus, and select imagesthat have a diversity of poses and scales.) The booklet can include adistillation of textual comments from all of the images in thecollection. (Again, the distillation may be manually assembled, orautomated techniques can be used. For example, a collection of textualcomments can be machine-analyzed to identify redundant words andphrases. The most repeated words/phrases are likely the most relevantdescriptors of the sailboat and the event. A few comments including therepeated words/phrases can be prioritized among the comment metadata,while other repetitions can be removed. The automated technique can alsoidentify infrequently used words/phrases, and include these aswell—since these may be particular to that sailboat and event.) Thebooklet can be formatted (by human direction, or automatically) printthe distilled comments around the margins of the selected pictures.

The foregoing examples are techniques based on semantic proximity.

In the examples just-given, and in others, it may be appropriate toscore metadata for relevance. This can determine, e.g., the order (orother presentation) by which such information is displayed to users.Thus, for example, if images are discerned to be related due to temporalproximity, an image that is closest in time to the subject image willusually be given a higher relevance score than another image that ismore remote in time Likewise, metadata for the closest-in-time imagewill typically be scored higher in relevance than metadata for a moreremote-in-time image. In a particular embodiment, a measure of semanticproximity is computed, and is used as the relevance score.

(An image extracted from a video can be watermarked to associate thevideo with the image—including identifying where in the video the imagecan be found. In some embodiments, a printer can work with a videofile—extracting one or more images for printing (perhaps composited fromseveral frames of video imagery)—and watermarking/registering each.)

The watermark identifiers can be assigned sequentially. More typical,however, is for identifiers to be structured with different fields, orranges. For example, some bits—or value ranges—may be used to indicatewhich registry issued the identifier, which device (or whose device)requested issuance of the identifier, etc.

The foregoing description particularly considers mobile phoneinteraction with watermarked documents. However, other devices can beused, including the scanner portion of multi-function (scan/print/fax)devices, and game systems that include a camera component and networkconnectivity (e.g., the Nintendo dSI handheld game system). For example,a game system may be configured to link from a printed picture to anonline blog about the subject of picture—to which viewers are invited tosubmit comments (which will then be presented to later viewers of thepicture). A photo taken at a school prom, for example, may link to ablog of viewers' recollections about the event.

In some implementations, before requesting a watermark identifier fromthe registry (e.g., for embedding in an image to be printed), the imageis first checked for the presence of a previous watermark. If one isfound, the system may skip the requesting and watermarking operations.Alternatively, the system may request a watermark anyway, and embed itas a second watermark in the image. (The system may report the previouswatermark to the registry, so the registry database can reflect that atleast certain hardcopies bearing the previous watermark also bear thenew watermark, and that all hardcopies bearing the new watermark alsobear the previous watermark.)

If an image is to be re-sized for printing (or its resolution is to bechanged), and it is found to have a pre-existing watermark, thewatermark embedder may remove the pre-existing watermark, re-size theimage, and then re-embed the re-sized image with a watermark identifier,i.e., the identifier formerly encoded by the removed watermark, oranother one. (Some watermarks are straightforward to remove, as isfamiliar to artisans in the field. This generally involves extractingthe watermark payload, creating a new pattern encoding that payload, andthen subtracting that pattern from the encoded image (as contrasted withthe usual addition of that pattern to an unencoded image)).

Although the detailed embodiments have focused on hardcopy photographs,the same principles are equally applicable to other printed documents(e.g., text), as well as to other forms of content (e.g., imagerydisplayed on theatrical screens and on electronic displays, etc.).

While particularly described in connection with image watermarking,similar functionality can be achieved by barcode-marking of imagery(albeit with impaired aesthetics). Likewise, image-fingerprinting can beused in lieu of watermarking if desired (albeit with less certain imageidentification).

Hybrid approaches can also be used, with part of the identificationachieved by one means (e.g., watermarking), and part by another (e.g.,image fingerprinting). For example, a watermark identifier may serve toassociate a photograph with a particular creator, and fingerprint datamay be used to identify a particular photograph within that creator'scollection (and associate that photograph with corresponding metadata).Such arrangement greatly simplifies the problem of fingerprint matchingthat is onerous in other applications, since a relatively modest,bounded set of images—on the order of 10,000 or so—is involved.

If several images are visually similar, their fingerprints will besimilar. Similar images identified in this fashion, and/or theirmetadata, may be presented for the user to select from. In the exampleof the sailboat pictures reviewed earlier, for example, imaging onehardcopy print may trigger an online experience (e.g., return ofmetadata) for that particular picture, with the software also inquiringwhether the user would like to also see the responses for visuallysimilar images.

Review

As will be recognized, the present disclosure details a great variety ofinventive subject matter. A small subject of such subject matter isreviewed in the following paragraphs.

One inventive arrangement is a method in which an issuing authorityreceives a request that the authority assign an identifier for use witha first image, so the assigned identifier can later be encoded in thefirst image by digital watermarking. First fingerprint datacorresponding to this first image is obtained, and a fingerprint datastore is checked to identify a second image having fingerprint data thatcorresponds with said first fingerprint data, for which an identifierhas already been issued. (As with all fingerprint techniques, exactidentity of fingerprints is rare. The required degree of correspondencecan be set by a numeric threshold, such as a Euclidean distancemeasure.)

This method further including either: (a) assigning, for use with thefirst image, the identifier earlier issued for use with the secondimage; or (b) assigning, for use with the first image, a new identifier.In this later case, a database record associated together the firstidentifier and the identifier earlier issued for use with the secondimage, so that the database indicates that the first content object andthe former content object have fingerprint data that correspond.

The assigned identifier may be encoded in the first image by digitalwatermarking. Alternatively, another data conveyance mechanism, such asa barcode or an RFID chip, can be employed. The referenced fingerprintmay comprise facial recognition data.

Another method involves receiving a request (e.g., at an issuingauthority) to assign an identifier for use with a first content object(e.g., a first image). The identifier is assigned, and transmitted to aremote device. The method further includes checking to determine if anidentifier has earlier been assigned to another content object having aperceptual similarity with the first content object.

If such checking determines that an identifier was earlier assigned,then the identifier assigned to the first content object can beassociated together in a database with the identifier earlier assignedto the other content object, so that the database indicates that thefirst content object and the former content object have a perceptualsimilarity. Alternatively, the earlier-assigned identifier may beassigned (again) as the first identifier. With this latter approach,both the first content object and the former content object are assignedthe same identifier.

In some embodiments, the checking is limited to content objects that areassociated with the same proprietor. That is, if User A requests anidentifier for user with the first content object, then only othercontent objects having User A as their proprietor are checked forperceptual similarity.

(The registry database can store, for each assigned identifier, dataidentifying the original proprietor of that content object. This datacan comprise, e.g., a hash of the user's email address, phone number,and/or device MAC address, or other such identifying information.Devices associated with a particular proprietor can be similarlyidentified, and data associating such devices with the proprietor can bestored—in the registry database or elsewhere.)

In some implementations, the request for an identifier is accompanied bythe content object itself (or a reduced resolution object, e.g., athumbnail image). In such case, the receiving system may compute afingerprint to check perceptual similarity. In other implementations, nocontent object is submitted with the request for an identifier (in whichcase fingerprint data may be sent with the request.)

Another method involves use of a hardware processor to decode anidentifier from imagery captured by a mobile phone from a first printedor displayed image. (The hardware processor may be part of the mobilephone, or not.) This decoded identifier is transmitted to a registry.Through a user interface, the system receives from a user anidentification of a second item of content with which the first imageryshould be associated, where the second item of content includes adifferent image, video or audio content item. Information about thissecond item of content is then transmitted to the registry. Thislate-binding arrangement enables the first image to link to the seconditem of content, by reference to the registry.

Another method is similar, but involves the viewer side of theexperience. A hardware processor decodes a first identifier from imagerycaptured from a first printed or displayed image by a mobile phone. Thisdecoded first identifier is transmitted to a registry. As a consequenceof such transmitting, the method further includes receiving data relatedto a second content item, where the second content item includes adifferent image, video or audio content item. (This second content itemmay be watermarked, with an identifier different than the firstidentifier.) Information based on such received data (e.g., the secondcontent item itself, or related data) is then presented to a userthrough the mobile phone.

Still another arrangement is a system that includes a computer, amemory, and an interface for exchanging data with users. The memoryincludes a data structure associating a first image identifier withrelated information. More particularly, the data structure includesfirst information expressing a relationship between the first image anda different second image, where this relationship was indicated by afirst user via data received through the interface from that first user.The data structure further includes second information expressing arelationship between the first image and a different third image, wherethis relationship was indicated by a second user via data receivedthrough the interface from that second user. The system is adapted tooutput, in response to a user's submission of the first imageidentifier, data relating to at least the second image. By sucharrangement, the system defines a social network of imagery, which canbe queried through the interface to enable exploration of imageryindicated—by different users—as being related to said first image.

This system may also be adapted to output—with the data relating to atleast the second image—an identification of the first user who indicateda relationship between the first image and the second image.

The foregoing arrangement may be made clearer by reference to FIGS. 5-7.In the depicted arrangement, User A snaps a picture of her GoldenRetriever puppy (“Milo”), and posts the picture to her Facebook page.She also prints the photo from the Facebook page, and puts the hardcopyprint on her refrigerator. (The print was steganographicallywatermarked—as detailed above—as a consequence of its printing, with thewatermark payload linking to User A's Facebook page.)

User B visits User A, and views the print from the refrigerator usinghis mobile phone. This encounter leads User B's mobile phone to User A'sFacebook page for the original image. User B recalls an image he'dearlier captured of Milo and his own dog Minnie, which he posted toFlickr. He gives Facebook a link to that photo (which Facebook thenimports onto the comment page). User B comments on the additional photowith the text, “Cute! Here's a picture I took of Milo and Minnie at thepark last week.”

User C thereafter surfs User A's Facebook page, and finds the originalMilo picture, as well as User B's added picture of Milo. User C adds acomment “Cute now, but they grow up and shed! Check out this link.” UserC then included a link to the Wikipedia page for Golden Retrievers.(That web page includes other images of that breed of dog.)

The Facebook web page then looks something like FIG. 6, with theoriginal photo of Milo, together with comments added by Users B and C,and the further image posted by User B, and the Wikipedia link added byUser C.

User D later snaps a picture of the hardcopy print on User A'srefrigerator, and posts it to a web site devoted to GoldenRetrievers—adding the tag “Sleeping beauty.”

User E encounters User D's posting on the Golden Retriever web site,prints the image of Milo, and posts this hardcopy on a bulletin board ather work. (User E's printer recognized that the image already included awatermark, so did not embed a further watermark.)

Turning now to FIG. 7, User F captures a mobile phone image of thehardcopy picture posted by User E on the work bulletin board. User F'smobile phone decodes the watermark identifier from the image (as encodedwhen User A printed the image for her refrigerator). The phone consultsthe registry to determine what auxiliary content should be presented inconnection with this image.

The registry includes a data structure, conceptually represented by thetable of FIG. 8. It associates various metadata with the identifieroriginally assigned when User A printed the first hardcopy(98D28D478C34). This table includes, in its first row, a UniformResource Identifier (URI) comprising a web link (URL) to the Facebookpage where User A originally posted the image of Milo. This web pageincludes the images, comments, and Wikipedia link shown in FIG. 6. Othermetadata in this row indicates a particular user associated with thismetadata (if known), and whether the URI points to a copy of theoriginal image.

Each time an item of registry data is accessed in connection with a userview of a watermarked picture, some or all of the web destination(s)associated with that picture may be checked by the registry soonthereafter for the addition of other content or links that should beadded to the registry entry for that watermark identifier. For example,when User B accessed the Facebook page, by viewing the refrigeratorpicture with his mobile phone, this prompted the registry to thereaftercheck the Facebook page for updates. This leads to the discovery thatUser B has added a comment, and a Flickr photo link. Information gleanedfrom such review can be added to the registry.

The second row of the FIG. 8 table gives the Flickr URL for the image ofMilo and Minnie, which User B contributed for posting on User A'sFacebook page, and which the registry thereafter discovered.

Instead of checking the linked URIs only when accessed, the registry mayperiodically scan all of linked URIs noted in its database, to harvestany other metadata that might have been added in association with suchsites. By such a periodic scan (e.g., weekly), the registry discoveredUser C's annotation of the Facebook page with the “shed!” comment, andthe link to Wikipedia. Such information is added to the registry entryfor this watermark identifier, as shown in the third row of FIG. 8.

The fourth, fifth, and sixth rows give particular URLs for other GoldenLabrador images found in the Wikipedia article. The registry found thesefurther URLs when it checked the Wikipedia entry just-noted.

The last depicted row of the FIG. 8 table gives the URL to the GoldenRetriever fan web site, to which User D posted a copy of the Milopicture. This item of metadata was contributed to the registry databaseby an automated web crawler that continuously examines pages across theweb, identifies encoded watermarks, and reports its discoveries back tothe registry (or to another service, which reports at least some ofthese discoveries back to the registry).

The web crawler reported back to the registry a link to the Milo imagefound on the Golden Retriever fan site, together with the watermarkidentifier discerned from that image. The registry added thisinformation to its table for the Milo image. (The last column of thisrow indicates the URI points to a copy of the originally-watermarkedimage.)

Information from the FIG. 8 table is sent to User F's phone, whichenables the phone to present a wealth of information through which UserF can browse (e.g., by sweeping her finger across the phone'stouchscreen—each sweep presenting a new item of associated information).Among the items presented are text entries excerpted from User A's webpage, the image of Milo and Minnie contributed by User B, other GoldenLabrador images from the Wikipedia web page (as well as the web pageitself), etc.

Again, all of this content is compiled, and made available, withoutexpress user action. Yet the hardcopy print on User A's refrigerator ledto the rich set of related information later presented to User F. (UserF may add to the cycle, e.g., by posting further comments to User A'sweb page, or to one of the other online sites noted.)

Thus, another method involves decoding (e.g., using a programmedhardware processor) a first identifier (e.g., 98D28D478C34) from imagerycaptured by a mobile phone from a first printed or displayed image(e.g., the hardcopy of Milo posted on the work bulletin board). Thisfirst identifier is transmitted to a registry. As a consequence, linkingdata is returned. With this linking data, a mobile phone accesses atleast one remote store (e.g., Facebook, Flickr, Wikipedia) containingcontent (e.g., images, comments) related to the image, where thiscontent is compiled by one or more individuals (e.g., Users B, C, D)having an image-related connection back to a proprietor of the image. Atleast some of this content is rendered to a user of the mobile phone(e.g., User F). Thus, a crowd-sourced collection of content isidentified by reference to the first identifier.

CONCLUDING REMARKS

Particularly contemplated mobile phones include the Apple iPhone 4, andsmartphones following Google's Android specification (e.g., the VerizonDroid Eris phone, manufactured by HTC Corp., and the Motorola Droid 2phone).

(Details of the iPhone, including its touch interface, are provided inApple's published patent application 20080174570.)

The design of mobile phones and computers referenced in this disclosureis familiar to the artisan. In general terms, each includes one or moreprocessors, one or more memories (e.g. RAM), storage (e.g., a disk orflash memory), a user interface (which may include, e.g., a keypad, aTFT LCD or OLED display screen, touch or other gesture sensors, a cameraor other optical sensor, a compass sensor, a 3D magnetometer, a 3-axisaccelerometer, one or more microphones, etc., together with softwareinstructions for providing a graphical user interface), interconnectionsbetween these elements (e.g., buses), and an interface for communicatingwith other devices (which may be wireless, such as GSM, CDMA, W-CDMA,CDMA2000, TDMA, EV-DO, HSDPA, WiFi, WiMax, or Bluetooth, and/or wired,such as through an Ethernet local area network, a T-1 internetconnection, etc).

FIG. 3 shows a block diagram of a representative mobile phone. A blockdiagram of a representative printer is shown in FIG. 4. The two depicteddevices have many elements in common, e.g., a processor, a display, atouchscreen and other physical user interface (UI) elements (buttons,etc.). Each further includes a network adapter for TCP/IP or othercommunication with other devices. The depicted devices also eachincludes a wireless (radio) interface, and a GPS-based location sensingunit. Each device further includes a memory that stores operating systemsoftware (including user interface software), application softwaremodules, etc. Watermark encoding/decoding software can be included amongthese software modules.

Among the relatively few differences, the smartphone of FIG. 3 includesa camera and microphone, whereas the printer of FIG. 4 includes ascanner and a print engine.

While this specification earlier noted its relation to the assignee'sprevious patents and patent applications, it bears repeating. Thesedisclosures should be read in concert and construed as a whole.Applicants intend that features and implementation details in eachdisclosure be combined and used in conjunction with such teachings inthe others. Thus, it should be understood that the methods, elements andconcepts disclosed in the present application be combined with themethods, elements and concepts detailed in those related applications.While some have been particularly detailed in the present specification,many have not—due to the large number of permutations and combinations.However, implementation of all such combinations is straightforward tothe artisan from the provided teachings.

Elements and teachings within the different embodiments disclosed in thepresent specification are also meant to be exchanged and combined.

The processes and system components detailed in this specification maybe implemented as instructions for computing devices, including generalpurpose processor instructions for a variety of programmable processors,including microprocessors (e.g., the Atom and A4), graphics processingunits (GPUs, such as the nVidia Tegra APX 2600), and digital signalprocessors (e.g., the Texas Instruments TMS320 series devices), etc.These instructions may be implemented as software, firmware, etc. Theseinstructions can also be implemented in various forms of processorcircuitry, including programmable logic devices, field programmable gatearrays (e.g., the Xilinx Virtex series devices), field programmableobject arrays, and application specific circuits—including digital,analog and mixed analog/digital circuitry. Execution of the instructionscan be distributed among processors and/or made parallel acrossprocessors within a device or across a network of devices. Processing ofcontent signal data may also be distributed among different processorand memory devices. As noted, “cloud” resources can be used as well.References to “processors,” “modules” or “components” should beunderstood to refer to functionality, rather than requiring a particularform of implementation.

Software instructions for implementing the detailed functionality can beauthored by artisans without undue experimentation from the descriptionsprovided herein, e.g., written in C, C++, Visual Basic, Java, Python,Tcl, Perl, Scheme, Ruby, etc. Mobile phones, printers, and other devicesaccording to certain implementations of the present technology caninclude software modules for performing the different functions andacts.

Known browser software, communications software, and media processingsoftware can be adapted for many of the uses detailed herein.

The service by which users ascribe certain attributes and experiences tocontent (e.g., through invocation of specified software) typically usessoftware on the user device—either in the OS or as application software.Alternatively, this service can be implemented—in part—using remoteresources.

Software and hardware configuration data/instructions are commonlystored as instructions in one or more data structures conveyed bytangible media, such as magnetic or optical discs, memory cards, ROM,etc., which may be accessed across a network. Some embodiments may beimplemented as embedded systems—a special purpose computer system inwhich the operating system software and the application software isindistinguishable to the user (e.g., as is commonly the case in basiccell phones). The functionality detailed in this specification can beimplemented in operating system software, application software and/or asembedded system software.

As noted, different of the functionality can be implemented on differentdevices. For example, in a system in which a mobile phone communicateswith a server at a remote service provider, different tasks can beperformed exclusively by one device or the other, or execution can bedistributed between the devices. Extraction of fingerprint and watermarkdata from content is one example of a process that can be distributed insuch fashion. Thus, it should be understood that description of anoperation as being performed by a particular device (e.g., a mobilephone or printer) is not limiting but exemplary; performance of theoperation by another device (e.g., a remote server), or shared betweendevices, is also expressly contemplated.

(In like fashion, description of data being stored on a particulardevice is also exemplary; data can be stored anywhere: local device,remote device, in the cloud, distributed, etc.)

In actual practice, data structures used by the present technology maybe distributed. For example, different registries may be employed bydifferent users. A system may need to navigate a series of intermediatedata structures (often hierarchical) to locate the one with neededinformation. (One suitable arrangement is detailed in Digimarc's U.S.Pat. No. 6,947,571.) Commonly accessed information may be cached atservers in the network—much like DNS data—to speed access.

While this disclosure has detailed particular ordering of acts andparticular combinations of elements, it will be recognized that othercontemplated methods may re-order acts (possibly omitting some andadding others), and other contemplated combinations may omit someelements and add others, etc.

Although disclosed as complete systems, sub-combinations of the detailedarrangements are also separately contemplated.

Reference was made to image fingerprinting. Various techniques are knownto artisans in the field. One technique is robust hashing, e.g., asdetailed in U.S. Pat. No. 6,671,407. Another is SURF, e.g., as utilizedin the assignee's application 61/485,888, filed May 13, 2011. SURF isdetailed, e.g., in Bay et al, “SURF: Speeded Up Robust Features,” Eur.Conf. on Computer Vision (1), pp. 404-417, 2006; as well as Chen et al,“Efficient Extraction of Robust Image Features on Mobile Devices,” Proc.of the 6th IEEE and ACM Int. Symp. On Mixed and Augmented Reality, 2007;and Takacs et al, “Outdoors Augmented Reality on Mobile Phone UsingLoxel-Based Visual Feature Organization,” ACM Int. Conf. on MultimediaInformation Retrieval, October 2008.)

To provide a comprehensive disclosure, without unduly lengthening thisspecification, applicants incorporate-by-reference the patentapplications and other documents referenced herein. (Such materials areincorporated in their entireties, even if cited above in connection withspecific of their teachings.) These references disclose technologies andteachings that can be incorporated into the arrangements detailedherein, and into which the technologies and teachings detailed hereincan be incorporated. The reader is presumed to be familiar with suchprior work.

1. A method comprising: at an issuing authority, receiving a requestthat the authority assign an identifier for use with a first image, theassigned identifier later to be encoded in the first image by digitalwatermarking; obtaining first fingerprint data corresponding to thefirst image; checking a fingerprint data store to identify a secondimage having fingerprint data that corresponds with said firstfingerprint data, the second image earlier having been issued anidentifier for use therewith; the method further including either: (a)assigning, for use with the first image, the identifier earlier issuedfor use with the second image; or (b) assigning, for use with the firstimage, a new identifier, and associating together, in a database, thefirst identifier and the identifier earlier issued for use with thesecond image, so that the database indicates that the first contentobject and the former content object have fingerprint data thatcorrespond.
 2. The method of claim 1 that comprises act (a) instead ofact (b).
 3. The method of claim 1 that comprises act (b) instead of act(a).
 4. The method of claim 1 that further includes encoding theassigned identifier in the first image by digital watermarking.
 5. Themethod of claim 1 wherein said fingerprint data comprises facialrecognition data.
 6. A method comprising: receiving a request to assignan identifier for use with a first content object; assigning anidentifier for use with the first content object, and transmitting theassigned identifier to a remote device; and checking to determine if anidentifier has earlier been assigned to a former content object having aperceptual similarity with the first content object.
 7. The method ofclaim 6 that includes: determining that an identifier has earlier beenassigned to a former content object having a perceptual similarity withthe first content object; and associating together, in a database, anidentifier assigned to the first content object, and the identifierearlier assigned to the former content object, so that the databaseindicates that the first content object and the former content objecthave a perceptual similarity.
 8. The method of claim 6 that includes:determining that an identifier has earlier been assigned to a formercontent object having a perceptual similarity with the first contentobject; and assigning said earlier-assigned identifier as the firstidentifier, wherein both the first content object and the former contentobject are assigned the same identifier.
 9. The method of claim 6 inwhich the first content object is associated with a first proprietor,and the checking comprises checking to see if an identifier has earlierbeen assigned to a former content object associated with said firstproprietor.
 10. The method of claim 6 that includes obtaining firstfingerprint data associated with the first content object, and in whichthe checking comprises checking a data store for correspondingfingerprint data associated with a former content object.
 11. The methodof claim 10 wherein said received request does not also include thefirst content object itself, and wherein the method further includesreceiving the fingerprint data corresponding to the first content objectfrom a remote device.
 12. The method of claim 10 wherein said receivedrequest includes the first content object itself, and wherein the methodfurther includes computing the fingerprint data corresponding to thefirst content object.
 13. The method of claim 6 wherein said fingerprintdata comprises facial recognition data.
 14. A method comprising: ahardware processor decoding an identifier from imagery captured by amobile phone from a first printed or displayed image; transmitting thedecoded identifier to a registry; through a user interface, receivingfrom a user an identification of a second item of content with which thefirst imagery should be associated, the second item of contentcomprising a different image, video or audio content item; andtransmitting information about the second item of content to theregistry; wherein the identifier thereafter enables linking from thefirst image to the second item of content, by reference to the registry.15. A method comprising: a hardware processor decoding a firstidentifier from imagery captured by a mobile phone from a first printedor displayed image; transmitting the decoded first identifier to aregistry; as a consequence of said transmitting, receiving data thatincludes data related to a second content item, the second content itemcomprising a different image, video or audio content item; andpresenting information based on such received data to a user using saidmobile phone.
 16. The method of claim 15 in which said receivingcomprises receiving data related to a second content item that isdigitally watermarked with an identifier different than said firstidentifier.
 17. The method of claim 16 that includes receiving saidsecond content item, digitally watermarked with said identifierdifferent than said first identifier, and presenting said second contentitem to the user.
 18. A system for use in connection with an imageidentifier for a first image, the system comprising: a computer and amemory, the memory including a data structure that associatesinformation with said image identifier; an interface for exchanging datawith users; the data structure including first information expressing arelationship between the first image and a different second image, saidrelationship having been indicated by a first user via data receivedthrough the interface from said first user; the data structure includingsecond information expressing a relationship between the first image anda different third image, said relationship having been indicated by asecond user via data received through the interface from said seconduser; the system being adapted to output, in response to a user'ssubmission of the first image identifier, data relating to at least thesecond image; wherein the system defines a social network of imagery,which can be queried through the interface to enable exploration ofimagery indicated—by different users—as related to said first image. 19.The system of claim 18 in which the system is adapted to output, withsaid data relating to at least the second image, an identification ofthe first user who indicated a relationship between the first image andthe second image.
 20. A method comprising: a hardware processor decodinga first identifier from imagery captured by a mobile phone from a firstprinted or displayed image; transmitting the decoded first identifier toa registry; as a consequence of said transmitting, receiving linkingdata; with said mobile phone and said received linking data, accessingat least one remote store containing content related to the image, saidcontent compiled by one or more individuals having an image-relatedconnection back to a proprietor of said image; and rendering at leastsome of said content to a user of the mobile phone; wherein acrowd-sourced collection of content is identified by reference to saidfirst identifier.